Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for multi-run graphing in ilab / Crucible backend #120

Closed
wants to merge 29 commits into from

Conversation

dbutenhof
Copy link
Collaborator

Type of change

  • Refactor
  • New feature
  • Bug fix
  • Optimization
  • Documentation Update

Description

This draft PR is backed by PR #117 to add support for overlaying metric graphs across multiple runs. It'll be rebased on main and opened for review after #117 is merged.

Related Tickets & Documents

Primarily PANDA-600.

Checklist before requesting a review

  • I have performed a self-review of my code.
  • If it is a core feature, I have added thorough tests.

Testing

InstructLab CPT is using a persistent Crucible controller system in RDU3, tied to a 4-way L40S test system. The data store (a private OpenSearch instance) contains a set of Crucible runs capturing both training and SDG runs.

dbutenhof and others added 29 commits September 27, 2024 16:30
GET localhost:8000/api/v1/ilab/runs?benchmark=ilab will query the
ilab.crucible OpenSearch instance and return a list of ilab benchmark
runs.
Add Crucible readme file.

Cleanups and refactoring
Also added the option to override the default graph title generator using the
new `Graph.title` field.
This cleans up my direct API call to get the run's periods for graphing, to
use a separate action and a reducer.

I also experimented with trying to improve error diagnosis by looking at some
of the error responses to "toast" instead of just saying something went wrong.
Add a Crucible `close` method, and use a FastAPI yield dependency to ensure
every API connection is closed cleanly.
+ other review feedback
+ add some method documentation
+ misc review feedback
Multigraph API failed if more than one `Graph` element specified the same run;
fix to be smarter about missing run IDs.

This also contains experimental code to expose per-iteration param values,
which doesn't quite work but doesn't seem to hurt anything.
(And `/api/v1/ilab/runs` reports iterations in numerical order.)
Move the legend up off the graph (although 1.5 is arbitrary and maybe not
ideal -- is there a more dynamic way to do this).

Also, I fixed the name of a `.less` class earlier, but just happened to notice
the corresponding use...
Move "unique parameters" accordion back up with parameters using the new
expansion toggles.
@dbutenhof
Copy link
Collaborator Author

Subsumed by the new #125 and #127

@dbutenhof dbutenhof closed this Oct 25, 2024
@dbutenhof dbutenhof deleted the multirun branch October 25, 2024 11:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants